Contextually adaptive digital pathology interface

ABSTRACT

A method, system, and computer program product for an image visualization system (120) that includes a contextually adaptive digital pathology interface. At least one image of a biological sample stained for the presence of one or more biomarkers is obtained (300). The image is displayed on a display screen at a first zoom level (310), in which a first subset of user selectable elements is contemporaneously displayed (320). As a result of user input, the image being is displayed at a second zoom level (330), in which a second subset of user selectable elements are contemporaneously displayed with the image (340). The one or more elements within the second subset of user selectable elements are disabled or hidden at the first zoom level, or one or more elements within the first subset of user selectable elements are disabled Or hidden at the second zoom level.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 18/065,969, filed on Dec. 14, 2022, which is a continuation of U.S.application Ser. No. 17/272,125, filed on Feb. 26, 2021, which is theU.S. National Stage of International Application No. PCT/US2019/048060filed on Aug. 26, 2019, which claims the benefit of U.S. PatentProvisional Application No. 62/726,081, filed Aug. 31, 2018, thecontents of which are incorporated herein by reference in their entiretyfor all purposes.

BACKGROUND OF THE DISCLOSURE

Digital pathology refers to the management and interpretation ofpathology information in a digital environment. Scanning devices areused to image slides of biological samples, which may be stained, suchthat digital slides, e.g., whole slide images, are generated. Digitalpathology software enables digital slides to be stored in a computermemory device, viewed on a computer monitor, and analyzed for pathologyinformation.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure provides a graphical user interface adapted toprovide visualizations of images of biological samples stained for thepresence of one or more biomarkers, and to enable the display ofcontextually relevant elements (e.g. analysis tools, viewer panels) to auser based on whether pre-established criteria are met. In someembodiments, contextually relevant elements are provided to the userbased on a selected zoom level or a selection of an image having aparticular stain. It is believed that by providing only those elementsthat are contextually relevant, the user is not overwhelmed withvisualized elements (e.g. those that are not appropriate or those thatwould be ineffective in a given context) and can instead select thecorrect elements based on a condition state (e.g. a selected zoom level,a selected image type, a selected tissue type, a selection of a wholeslide image versus a slide derived from a tissue microarray, a selectedslide having a particular stain applied (for instance, H&E), a selectedslide stained for the presence of a particular biomarker, a selectedslide stained using immunohistochemistry as opposed to in situhybridization, a selected slide that has had certain image analysisalgorithms applied). It is further believed that providing only elementsthat are contextually relevant allows for a quicker and more accuratereview of presented image data. In addition, given that display screenshave a limited size and resolution, it is often difficult to providemultiple visualizations at once or if multiple visualizations areprovided, they may overlap or obstruct a user's view of importantelements. By displaying only those elements that are contextuallyrelevant given a particular condition (e.g. a particular zoom levelselected by a user), the most appropriate elements may be prioritizedand visualized taking into account the limited display size andresolution.

In some embodiments, a computing device comprises a display screen, thecomputing device configured to: access at least one image of abiological sample stained for the presence of one or more biomarkers(e.g. HER2, PD-L1, etc.) from one or more memories communicativelycoupled to the computing device; display on the display screen a firstvisualization of the at least one image at a first zoom level, andcontemporaneously with displaying the first visualization, display onthe display screen a first subset of user selectable elements; andsubsequently display on the display screen a second visualization of theat least one image at a second zoom level, wherein the second zoom levelis greater than the first zoom level, and contemporaneously withdisplaying the second visualization, display on the display screen asecond subset of user selectable elements, (i) one or more elementswithin the second subset of user selectable elements are disabled orhidden at the first zoom level, or (ii) one or more elements within thefirst subset of user selectable elements are disabled or hidden at thesecond zoom level.

In some embodiments, the second zoom level at least meets apre-determined threshold zoom value. In some embodiments, the secondsubset of user selectable elements are contextually relevant at thesecond zoom level. In some embodiments, the user selectable elementscomprise menu bar tools and contextual menu items. By way of example,the tools and menus are contextual. When a user is fully “zoomed out,”only the relevant tools to use in that context are enabled (see FIG.4A). When the user zooms in to a specified zoom level (see FIG. 4B),tools in the tool bar become enabled and/or contextual panels appear.The user can now use these tools when in the “zoomed in” context of theslide. As such, the tools available in either a first series of tools orin a second series of tools depends on the zoom level selected by auser, e.g. a second zoom level compared to a first zoom level, whereinthe second zoom level meets or exceeds a pre-determined threshold zoomlevel for enabling the second series of tools.

In some embodiments, the menu bar tools comprise image annotation tools,slide setup tools, slide selection tools, navigation tools and viewingtools. In some embodiments, the image annotation tools are enabled atthe second zoom level. In some embodiments, the image annotation toolsare hidden at the first zoom level. In some embodiments, the imageannotation tools comprise region of interest identification tools,measurement tools, indicia drawing tools, and region exclusion tools.

In some embodiments, the computing device is further configured todisplay on the display screen one or more viewer panels at the secondzoom level, wherein the one or more viewer panels for display at thesecond zoom level are disabled at the first zoom level. In someembodiments, the first zoom level is a 1× zoom level (i.e. the lowestzoom level, such as an image scanned or otherwise captured at nomagnification) and wherein the first visualization comprises at leastone representation including the at least one image of the biologicalsample. In some embodiments, the at least one representation furthercomprises a portion comprising identifying indicia.

In some embodiments, a computing device comprises a display screen, thecomputing device configured to: access at least one image of abiological sample stained for the presence of one or more biomarkersfrom one or more memories communicatively coupled to the computingdevice; display on the display screen at least a first representationcomprising the at least one image at a first zoom level, andcontemporaneously with displaying the first representation, display onthe display screen at least a first viewer panel; subsequently displayon the display screen a second representation of the at least one imageat a second zoom level, wherein the second zoom level is greater thanthe first zoom level, and contemporaneously with displaying the secondrepresentation, display on the display screen at least a second viewerpanel in addition to the first viewer panel, wherein the at least thesecond viewer panel is hidden from display at the first zoom level.

In some embodiments, a third viewer panel is displayed on the displayscreen contemporaneously with the displaying of the second displaypanel, wherein the third viewer panel is hidden from display at thefirst zoom level. In some embodiments, menu bar icons are enabled on thedisplay screen contemporaneously with the displaying of the seconddisplay panel. In some embodiments, the menu bar icons displayed areselected from the group consisting of region of interest identificationtools, measurement tools, indicia drawing tools, and region exclusiontools. In some embodiments, the at least the first representationcomprises: (i) a first portion comprising the at least one image of thebiological sample at the first zoom level, and (ii) a second portioncomprising identifying indicia. In some embodiments, the identifyingindicia comprises an identification of a biomarker.

In some embodiments, a method comprises: displaying, on a computingdevice having a display screen, a first visualization comprising atleast on image of a stained biological sample, wherein the firstvisualization is displayed in a first condition state, andcontemporaneously with displaying, the first visualization, display onthe display screen at least one of a first viewer panel or a firstseries of user selectable elements; and subsequently display on thedisplay screen a second visualization of the at least one image in asecond condition state, wherein the second condition state is derived asa result of a user selection, and contemporaneously with displaying thesecond visualization, display on the display screen at least one of asecond series of user selectable elements that are not enabled in thefirst condition state or a second viewer panel in addition to the firstviewer panel, wherein the second viewer panel is hidden from display atthe first zoom level. In some embodiments, the first condition state isa default state. In some embodiments, the user selection is a zoomlevel, and wherein the second condition is a zoom level greater than thedefault zoom level. In some embodiments, the user selection is an imageselection, wherein the image selected for the second condition comprisesa different gain than an image of the first condition state.

In some embodiments, a method comprises: displaying, on a computingdevice having a display screen, a first visualization comprising atleast one image of a stained biological sample, wherein the firstvisualization is displayed at a first zoom level, and contemporaneouslywith displaying the first visualization, display on the display screenat least one of a first viewer panel or a first series of userselectable elements; and subsequently display on the display screen asecond visualization of the at least one image in at a second zoomlevel, wherein the second zoom level is greater than the first zoomlevel, and contemporaneously with displaying the second visualization,display on the display screen at least one of a second series of userselectable elements that are hidden or not enabled at the first zoomlevel or at least a second viewer panel in addition to the first viewerpanel, wherein the second viewer panel is hidden from display at thefirst zoom level.

In some embodiments, the first zoom level is a default zoom level. Insome embodiments, the second zoom level is one which sufficientlyresolves a cluster of cells or which sufficiently resolves cell nuclei.In some embodiments, the first zoom level is a lowest available zoomlevel and the second zoom level is at least 5×. In some embodiments, thefirst zoom level is a lowest available zoom level and the second zoomlevel is at least 10×.

In some embodiments, the second subset of user selectable elements arecontextually relevant at the second zoom level. In some embodiments, theuser selectable elements comprise menu bar tools and contextual menuitems. In some embodiments, the menu bar tools comprise image annotationtools, slide setup tools, slide selection tools, navigation tools andviewing tools. In some embodiments, the image annotation tools areenabled at the second zoom level. In some embodiments, the imageannotation tools are hidden at the first zoom level. In someembodiments, the image annotation tools comprise region of interestidentification tools, measurement tools, indicia drawing tools, andregion exclusion tools.

In some embodiments, the first visualization includes at least onerepresentation comprising: (i) a first portion comprising the at leastone image of the biological sample at the first zoom level, and (ii) asecond portion comprising identifying indicia. In some embodiments, theidentifying indicia comprises an identification of a biomarker.

In some embodiments, a non-transitory computer-readable medium storesinstructions which, when executed by one or more processors of acomputing system, causes the computing system to display on the displayscreen a first visualization of the at least one image at a first zoomlevel, and contemporaneously with displaying the first representation,display on the display screen a first subset of user selectable tools;and subsequently display on the display screen a second visualization ofthe at least one image at a second zoom level, wherein the second zoomlevel is greater than the first zoom level, and contemporaneously withdisplaying the second representation, display on the display screen asecond subset of user selectable tools, wherein one or more tools withinthe second subset of user selectable tools are not enabled at the firstzoom level.

BRIEF DESCRIPTION OF THE FIGURES

For a general understanding of the features of the disclosure, referenceis made to the drawings. In the drawings, like reference numerals havebeen used throughout to identify identical elements.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided to the Office upon request and thepayment of the necessary fee.

FIG. 1 illustrates a system including a computer having one or moreprocessors and a scanning device, where the computer and the scanningdevice are communicatively coupled, such as through a network, inaccordance with some embodiments.

FIG. 2A illustrates a system including a processing subsystem, a storagesubsystem, an output device, and an input device, each of the componentscommunicatively coupled through a bus, a network, or other wired orwireless interconnect, in accordance with some embodiments. The systemmay also include software to enable remote access, i.e. a client portalor client interface.

FIG. 2B sets forth a block diagram of a system communicatively coupledwith a client interface through a network in accordance with someembodiments.

FIG. 3 sets forth a flowchart providing the general steps of displayingvisualizations in accordance with some embodiments.

FIG. 4A illustrates a viewer window comprising a visualization includinga plurality of representations at a first zoom level in accordance withsome embodiments.

FIG. 4B illustrates a viewer window comprising a visualization includingrepresentation at a second zoom level, the second zoom level beinggreater than the first zoom level of FIG. 4A in accordance with someembodiments.

FIG. 4C illustrates a viewer window comprising a visualization includinga plurality of representations at a third zoom level, the third zoomlevel being between the first and second zoom levels of FIGS. 4A and 4B,respectively, in accordance with some embodiments.

FIG. 5A illustrates a dropdown menu for the selection of certain viewerpanels, such as those available at a first zoom level (e.g. the zoomlevel in FIG. 4A), in accordance with some embodiments.

FIG. 5B illustrates a dropdown menu for the selection of certain viewerpanels, such as those available at a second zoom level (e.g. the zoomlevel in FIG. 4B), in accordance with some embodiments.

FIG. 6A illustrates a viewer window comprising a first visualizationincluding three representations at a first zoom level, in accordancewith some embodiments.

FIG. 6B illustrates a viewer window comprising a first visualizationincluding three representations at a second zoom level, an intermediatezoom level as compared with zoom levels depicted in FIGS. 6A and 6C, inaccordance with some embodiments.

FIG. 6C illustrates a viewer window comprising a first visualizationincluding three representations at a third zoom level, the third zoomlevel being greater than the zoom level in FIGS. 6A and 6B, inaccordance with some embodiments.

DETAILED DESCRIPTION

It should also be understood that, unless clearly indicated to thecontrary, in any methods claimed herein that include more than one stepor act, the order of the steps or acts of the method is not necessarilylimited to the order in which the steps or acts of the method arerecited.

As used herein, the singular terms “a,” “an,” and “the” include pluralreferents unless context clearly indicates otherwise. Similarly, theword “or” is intended to include “and” unless the context clearlyindicates otherwise. The term “includes” is defined inclusively, suchthat “includes A or B” means including A, B, or A and B.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of or “exactly one of,” or, when used inthe claims, “consisting of” will refer to the inclusion of exactly oneelement of a number or list of elements. In general, the term “or” asused herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of” “only one of” or“exactly one of” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

The terms “comprising,” “including,” “having,” and the like are usedinterchangeably and have the same meaning. Similarly, “comprises,”“includes,” “has,” and the like are used interchangeably and have thesame meaning. Specifically, each of the terms is defined consistent withthe common United States patent law definition of “comprising” and istherefore interpreted to be an open term meaning “at least thefollowing,” and is also interpreted not to exclude additional features,limitations, aspects, etc. Thus, for example, “a device havingcomponents a, b, and c” means that the device includes at leastcomponents a, b and c. Similarly, the phrase: “a method involving stepsa, b, and c” means that the method includes at least steps a, b, and c.Moreover, while the steps and processes may be outlined herein in aparticular order, the skilled artisan will recognize that the orderingsteps and processes may vary.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in some embodiments, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in some embodiments, to at least one, optionally including morethan one, B, with no A present (and optionally including elements otherthan A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

As used herein, the term “image data” encompasses raw image dataacquired from the biological tissue sample, such as by means of anoptical sensor or sensor array, or pre-processed image data. Inparticular, the image data may comprise a pixel matrix.

As used herein, the terms “image,” “image scan,” or “scanned image”encompasses raw image data acquired from the biological tissue sample,such as by means of an optical sensor or sensor array, or pre-processedimage data. In particular, the image data may comprise a pixel matrix.

As used herein, the term “biological sample,” “tissue sample,”“specimen” or the like refers to any sample including a biomolecule(such as a protein, a peptide, a nucleic acid, a lipid, a carbohydrate,or a combination thereof) that is obtained from any organism includingviruses. Other examples of organisms include mammals (such as humans;veterinary animals like cats, dogs, horses, cattle, and swine; andlaboratory animals like mice, rats and primates), insects, annelids,arachnids, marsupials, reptiles, amphibians, bacteria, and fungi.Biological samples include tissue samples (such as tissue sections andneedle biopsies of tissue), cell samples (such as cytological smearssuch as Pap smears or blood smears or samples of cells obtained bymicrodissection), or cell fractions, fragments or organelles (such asobtained by lysing cells and separating their components bycentrifugation or otherwise). Other examples of biological samplesinclude blood, serum, urine, semen, fecal matter, cerebrospinal fluid,interstitial fluid, mucous, tears, sweat, pus, biopsied tissue (forexample, obtained by a surgical biopsy or a needle biopsy), nippleaspirates, cerumen, milk, vaginal fluid, saliva, swabs (such as buccalswabs), or any material containing biomolecules that is derived from afirst biological sample. In certain some embodiments, the term“biological sample” as used herein refers to a sample (such as ahomogenized or liquefied sample) prepared from a tumor or a portionthereof obtained from a subject.

As used herein, the term “slide” refers to any substrate (e.g.,substrates made, in whole or in part, glass, quartz, plastic, silicon,etc.) of any suitable dimensions on which a biological specimen isplaced for analysis, and more particularly to a “microscope slide” suchas a standard 3 inch by 1-inch microscope slide or a standard 75 mm by25 mm microscope slide. Examples of biological specimens that can beplaced on a slide include, without limitation, a cytological smear, athin tissue section (such as from a biopsy), and an array of biologicalspecimens, for example a tissue array, a cellular array, a DNA array, anRNA array, a protein array, or any combination thereof. Thus, in someembodiments, tissue sections, DNA samples, RNA samples, and/or proteinsare placed on a slide at particular locations. In some embodiments, theterm slide may refer to SELDI and MALDI chips, and silicon wafers.

As used herein, the terms “stain,” “staining,” or the like as usedherein generally refers to any treatment of a biological specimen thatdetects and/or differentiates the presence, location, and/or amount(such as concentration) of a particular molecule (such as a lipid,protein or nucleic acid) or particular structure (such as a normal ormalignant cell, cytosol, nucleus, Golgi apparatus, or cytoskeleton) inthe biological specimen. For example, staining can provide contrastbetween a particular molecule or a particular cellular structure andsurrounding portions of a biological specimen, and the intensity of thestaining can provide a measure of the amount of a particular molecule inthe specimen. Staining can be used to aid in the viewing of molecules,cellular structures and organisms not only with bright-fieldmicroscopes, but also with other viewing tools, such as phase contrastmicroscopes, electron microscopes, and fluorescence microscopes. Somestaining performed by the system can be used to visualize an outline ofa cell. Other staining performed by the system may rely on certain cellcomponents (such as molecules or structures) being stained without orwith relatively little staining other cell components. Examples of typesof staining methods performed by the system include, without limitation,histochemical methods, immunohistochemical methods, and other methodsbased on reactions between molecules (including non-covalent bindinginteractions), such as hybridization reactions between nucleic acidmolecules. Particular staining methods include, but are not limited to,primary staining methods (e.g., H&E staining, Pap staining, etc.),enzyme-linked immunohistochemical methods, and in situ RNA and DNAhybridization methods, such as fluorescence in situ hybridization(FISH).

As used herein, the term “user interface” refers to the interface thatallows the user, for example end users such as histologists and/orpathologists, to input commands and data and receive results, such as agraphical user interface (GUI). The terms “user interface” and“graphical user interface” are used interchangeably herein.

As described in further detail herein, the present disclosure isdirected to a graphical user interface which allows a user to viewand/or analyze one or more images of biological samples stained for thepresence of one or more biomarkers, whereby the visualization of certainelements and/or the availability of certain analysis and/or processingtools are provided to the user on a contextual basis. In someembodiments, the graphical user interface is adapted to providecontextually relevant visualizations and elements for selection based onwhether a pre-established condition has been met. In some embodiments,the pre-established condition is a selection made by a user. Forexample, a selection may be a zoom level, or a magnification levelselected by a user, where different zoom level or magnification levelselections cause the graphical user interface to adaptively generatecertain visualizations or to enable certain user configurable items,e.g. annotation tools, image processing tools, etc. In some embodiments,the graphical user interface adjusts those visualizations, analysistools and/or viewer panels which are provided to the user (such as fordisplay on a display) depending on a selected zoom level, e.g. a firstsubset of contextually relevant tools may be presented to the user at alowest zoom level (e.g. no optical magnification as compared with, forexample, a 10×, 20×, or 40× optical magnification), while a moreinclusive second subset of tools may be presented to the user at agreater zoom level of 10× where the additional tools included within thesecond subset are again contextually relevant for the 10× zoom level. Byway of another example, a selection may be a particular type of tissue(i.e. an image of a tissue) selected by the user or an image of a tissuesample stained for the presence of a particular biomarker.

It is believed that by presenting only those visualizations, analysistools, viewer panels, etc. that are contextually relevant to apre-established condition, e.g. at a selected zoom level, a selectedtissue type, a selected slide having a particular stain applied, aselected slide stained for the presence of a particular biomarker, aselection of a whole slide image as opposed to a tissue microarray,etc., a user may be able to interact more efficiently with the software.Said another way, since the user is not overwhelmed with theavailability of a multitude tools and/or viewer panels that are notrelevant at a particular zoom level, the operator may be able tointeract more efficiently with the graphical user interface, ultimatelyresulting in a quicker review and analysis of the images of the tissuesamples presented.

In some embodiments, systems of the present disclosure are adapted tofacilitate the interpretation and reporting of image data obtained froma subject (e.g. a human patient). In some embodiments, image data isacquired from a scanning device (such as the VENTANA DP 200 scanner,available from Ventana Medical Systems, Inc., Tucson, AZ) and the imagedata may be stored in a database, such as a networked database, forlater visualization and analysis. For example, image data may beacquired using a scanning device and the scanned image data stored in afile located on a storage subsystem 104 or a networked server, wherebythe file may be later retrieved for visualization and analysis (see FIG.1 ). In some embodiments, software, such as an image visualization andanalysis application, is run directly on a system and the image data isretrieved from the networked server for interpretation and reporting bya user interacting with the software (see FIG. 2A). In some embodiments,software, such as an image visualization and analysis application, isrun on a remote system and a client interface or client portal is usedto access the system, and whereby image data may be retried from astorage subsystem for visualization and analysis (see FIG. 2B).

The systems and methods provided herein can be applied to thevisualization and analysis of any type of image of a tissue stained forthe presence of one or more biomarkers. For example, the biologicalsample may be stained through application of one or more stains, and theresulting image or image data comprises signals corresponding to each ofthe one or more stains. In some embodiments, the input images aresimplex images having only a single stain (e.g., stained with3,3′-diaminobenzidine (DAB)). In some embodiments, the biological samplemay be stained in a multiplex assay for two or more stains (thusproviding multiplex images). In some embodiments, the biological samplesare stained for at least two biomarkers. In some embodiments, thebiological samples are stained for the presence of at least twobiomarkers and also stained with a primary stain (e.g. hematoxylin). Insome embodiments, the biological samples are stained for the presence ofat least one protein biomarker and at least two nucleic acid biomarkers(e.g. DNA, RNA, microRNAs, etc.).

In some embodiments, the biological samples are stained in animmunohistochemistry assay for the presence of one or more proteinbiomarkers. For example, the biological sample may be stained for thepresence of a human epidermal growth factor receptor 2 protein (HER2protein). Currently in the United States, there are two Food and DrugAdministration (FDA) approved methods for HER2 assessment: HerceptTest™(DAKO, Glostrup Denmark) and HER2/neu (4B5) rabbit monoclonal primaryantibody (Ventana, Tucson, Arizona).

In some embodiments, the biological sample is stained for the presenceof estrogen receptor (ER), progesterone receptor (PR), or Ki-67. In yetother embodiments, the biological sample is stained for the presence ofEGFR or HER3. Examples of other protein biomarkers are described byZamay et. Al., “Current and Prospective Biomarkers of Long Cancer,”Cancers (Basel), 2018 November; 9(11), the disclosure of which is herebyincorporated by reference herein in its entirety. Examples of proteinbiomarkers described by Zamay include CEACAM, CYFRA21-1, PKLK, VEGF,BRAF, and SCC.

In some embodiments, the biological samples are stained in an in situhybridization (ISH) assay for the presence of one or more nucleic acids,including mRNA. U.S. Pat. No. 7,087,379 (the disclosure of which ishereby incorporated by reference herein in its entirety) describesmethods of staining samples with ISH probes such that individual spots(or dots), representing single gene copies, may be observed anddetected. In some embodiments, several target genes are simultaneouslyanalyzed by exposing a cell or tissue sample to a plurality of nucleicacid probes that have been labeled with a plurality of different nucleicacid tags.

FIG. 1 sets forth a system 100 (a computer or computing device)including an scanning device 110 communicatively coupled to a processingsubsystem 102. The scanning device 110 can be coupled to the processingsubsystem 102 either directly (e.g., through one or more communicationcables) or through one or more wired and/or wireless networks 130. Insome embodiments, the processing subsystem 102 may be included in orintegrated with the scanning device 110. In some embodiments, the system100 may include software to command the scanning device 110 to performcertain operations using certain user configurable parameters, and tosend resulting imaging data acquired to the processing subsystem 102 ora storage subsystem (e.g. a local storage subsystem or a networkedstorage device). In some embodiments, either the processing subsystem102 or the scanning device 110 may be coupled to a network 130. In someembodiments, a storage device is coupled to the network 130 for storageor retrieval of image data, subject information, and/or other tissuedata. The processing subsystem 102 may include a display 108 and one ormore input devices (not illustrated) for receiving commands from a useror operator (e.g. a technician, histologist, or pathologist).

In some embodiments, a user interface is rendered by processingsubsystem 102 and is provided on display 108 to (i) facilitate theanalysis, interpretation, and/or reporting of imaging data and/orsubject data; (ii) to retrieve data from a scanning device; or (iii) toretrieve imaging data, subject information, or other clinicalinformation from a database, such as one available through a network. Insome embodiments, the network 130 enables access to a processingsubsystem 102 and/or a scanning device 110 remotely, such as through aclient interface or client portal (not illustrated). In this way, aremote user may access the processing subsystem 102 such that imagevisualization and analysis software may be run remotely on theprocessing subsystem 102. In some embodiments, the client interface orclient portal may also enable the retrieval of stored reports afteranalysis of the imaging data.

FIG. 2A is a block diagram of a system 100 according to an embodiment ofthe present disclosure. System 100 can be implemented using any type ofuser-operable computing device, including desktop computers, laptopcomputers, tablet computers, handheld devices (e.g., smart phones, mediaplayers), and so on. System 100 can include a number of interconnectedcomponents such as processing subsystem 102, storage subsystem 104, userinput device 106, display 108, and network interface 112 communicatingvia bus 114, as discussed in more detail below. In some embodiments, thesystem 100 depicted in FIG. 2A may be accessed remotely, e.g. one ormore remote users may access system 100, such as over a network, suchthat image data stored within storage subsystem 104 may be reviewed,interpreted, analyzed, and/or reported.

Processing subsystem 102 can include a single processor, which can haveone or more cores, or multiple processors, each having one or morecores. In some embodiments, processing subsystem 102 can include one ormore general-purpose processors (e.g., CPUs), special-purpose processorssuch as graphics processors (GPUs), digital signal processors, or anycombination of these and other types of processors. In some embodiments,some or all processors in processing subsystem can be implemented usingcustomized circuits, such as application specific integrated circuits(ASICs) or field programmable gate arrays (FPGAs). In some embodiments,such integrated circuits execute instructions that are stored on thecircuit itself. In some embodiments, processing subsystem 102 canretrieve and execute instructions stored in storage subsystem 104, andthe instructions may be executed by processing subsystem 102 regardlessof whether a user accesses the system locally or remotely, such asthrough client portal 116. By way of example, processing subsystem 102can execute instructions to receive and process image data stored withina local or networked storage system and display the image data (e.g.display a whole slide scanned image, or a magnified portion of any wholeslide scanned image).

Storage subsystem 104 can include various memory units such as a systemmemory, a read-only memory (ROM), and a permanent storage device. A ROMcan store static data and instructions that are needed by processingsubsystem 102 and other modules of system 100. The permanent storagedevice can be a read-and-write memory device. This permanent storagedevice can be a non-volatile memory unit that stores instructions anddata even when system 100 is powered down. In some embodiments, amass-storage device (such as a magnetic or optical disk or flash memory)can be used as a permanent storage device. Other embodiments can use aremovable storage device (e.g., a flash drive) as a permanent storagedevice. The system memory can be a read-and-write memory device or avolatile read-and-write memory, such as dynamic random access memory.The system memory can store some or all of the instructions and datathat the processor needs at runtime.

Storage subsystem 104 can include any combination of non-transitorycomputer readable storage media including semiconductor memory chips ofvarious types (DRAM, SRAM, SDRAM, flash memory, programmable read-onlymemory) and so on. Magnetic and/or optical disks can also be used. Insome embodiments, storage subsystem 104 can include removable storagemedia that can be readable and/or writeable; examples of such mediainclude compact disc (CD), read-only digital versatile disc (e.g.,DVD-ROM, dual-layer DVD-ROM), read-only and recordable Blu-ray® disks,ultra-density optical disks, flash memory cards (e.g., SD cards, mini-SDcards, micro-SD cards, etc.), and so on. In some embodiments, image dataand/or subject data can be stored in one or more remote locations, e.g.,cloud storage, and synchronized with other the components of system 100.When the terms “memory” or “a memory” are used herein, they may refer toone or more memories, such as a plurality of memories.

In some embodiments, storage subsystem 104 can store one or moresoftware programs to be executed by processing subsystem 102, such as animage visualization and analysis application 120. “Software” refersgenerally to sequences of instructions that, when executed by processingsubsystem 102, cause system 100 to perform various operations, thusdefining one or more specific machine implementations that execute andperform the operations of the software programs. Thus, “software” canalso include firmware or embedded applications or any other type ofinstructions readable and executable by processing subsystem 102.Software can be implemented as a single program or a collection ofseparate programs or program modules that interact as desired. In someembodiments, programs and/or data can be stored in non-volatile storageand copied in whole or in part to volatile working memory during programexecution. From storage subsystem 104, processing subsystem 102 canretrieve program instructions to execute and data to process in order toexecute various operations including operations described below.

In some embodiments, the software may be run locally on system 100 butaccessed and/or controlled remotely, such as through a client portal116. For example, an instance of the image visualization and analysisapplication 120 may be run locally on system 100 but a remote operatormay access the image visualization and analysis application 120 by meansof a network connected client portal 116 such that a remote user maycontrol the instance of the image visualization and analysis application120 to facilitate the review, interpretation, and analysis of image data(e.g. a scanned image of a biological sample retrieved from the storagesubsystem 104 and presented to the remote user for analysis).

A user interface can be provided to a display 108, and/or and one ormore other user output devices (not shown). The user interface mayinclude, for example, visualizations and other representations, therepresentations including images derived from scans of stainedbiological samples (e.g. samples stained for the presence of one or morebiomarkers or stained with hematoxylin and eosin), menu bars, drop-downmenus, and/or panels. The user interface provided to the display may beadapted such that only contextually relevant tools and/or viewer panelsare provided to the user based on, for example, user selectionsincluding, but not limited to, a user selected zoom or magnificationlevel. User input devices 106 can include any device via which a usercan provide signals to system 100; system 100 can interpret the signalsas indicative of particular user requests or information. In someembodiments, user input devices 106 can include any or all of a keyboardtouch pad, touch screen (e.g., a touch-sensitive overlay on a displaysurface of display 108), mouse or other pointing device, scroll wheel,click wheel, dial, button, switch, keypad, microphone, and so on.

Display 108 can display visualizations (e.g. representations includingimage data, viewer panels to convey information to a user, or contextualmenus which provide user selectable configuration options, etc.)generated by system 100 and can include various image generationtechnologies, e.g., a cathode ray tube (CRT), liquid crystal display(LCD), light-emitting diode (LED) including organic light-emittingdiodes (OLED), projection system, or the like, together with supportingelectronics (e.g., digital-to-analog or analog-to-digital converters,signal processors, or the like). Some embodiments can include a devicesuch as a touchscreen that function as both input and output device. Insome embodiments, other user output devices can be provided in additionto or instead of display 108.

In some embodiments, the user interface can provide a graphical userinterface, in which visible image elements in certain areas of display108 are defined as active elements, interactive elements, or controlelements that the user selects using user input device 106. For example,the user can manipulate a user input device 106 to position an on-screencursor or pointer over the control element, then “click” a button toindicate the selection, with the selection sending signals to perform adesignated action or routine. For example, the user can manipulate theuser input device 106 to select an icon within the user interface (suchas one in a viewer panel, in a menu bar, or within a drop down menu),which would effectuate the initiation of an operation or selection of atool, e.g. initiate the annotation of one or more displayedrepresentations of an image of a tissue sample. By way of anotherexample, the user may click on a menu bar icon to initiate the selectionof a tool such that a region of interest may be selected by the userbased on received inputs. In some embodiments, the user can manipulatethe user input device 106 so as to interact with dropdown menus toselect one or more panels, including interactive panels. In someembodiments, these selections may only be made by the user if the toolsand/or viewer panels are enabled based on whether pre-establishedconditions are met, e.g. whether a certain zoom level is selected by auser, whether a particular tissue type is selected, whether a particularslide having a certain biomarker is selected.

Alternatively, the user can touch the control element (e.g., with afinger or stylus) on a touchscreen device. In some embodiments, the usercan speak one or more words associated with the control element (theword can be, e.g., a label on the element or a function associated withthe element). In some embodiments, user gestures on a touch-sensitivedevice can be recognized and interpreted as input commands; thesegestures can be, but need not be, associated with any particular area ondisplay 108. Other user interfaces can also be implemented.

Network interface 112 may provide data communication capability forsystem 100. In some embodiments, network interface 112 can include radiofrequency (RF) transceiver components for accessing wireless voiceand/or data networks (e.g., using cellular telephone technology,advanced data network technology such as 3G, 4G or EDGE, 5G, WiFi (IEEE802.11 family standards), or other mobile communication technologies, orany combination thereof, GPS receiver components, and/or othercomponents. In some embodiments, network interface 112 can provide wirednetwork connectivity (e.g., Ethernet) in addition to or instead of awireless interface. Network interface 112 can be implemented using acombination of hardware (e.g., antennas, modulators/demodulators,encoders/decoders, and other analog and/or digital signal processingcircuits) and software components. Network interface 112 may facilitateremote access to system 100, such as through a client portal 116 (forexample, a remote user may access system 100 through a remote computerand the remote computer interacts with system 100 through the networkinterface 112). In some embodiments, the client portal 116 is astand-alone application that is run by a remote user on a remotecomputer or other computing device. In some embodiments, the clientportal 116 is a web-browser running on the remote computer or othercomputing device which accesses system 100 through a network.

Bus 114 can include various system, peripheral, and chipset buses thatcommunicatively connect the numerous components of system 100. Forexample, bus 114 can communicatively couple processing subsystem 102with storage subsystem 104. Bus 114 can also connect to user inputdevices 106 and display 108. Bus 114 can also couple processingsubsystem 102 to a network through network interface 112. In thismanner, system 100 can be connected to a network of multiple computersystems (e.g., a local area network (LAN), a wide area network (WAN), anIntranet, or a network of networks, such as the Internet. The skilledartisan will appreciate that additional components may be connected tobus 114, such as a scanning device, a scanning device, a tissueprocessing system, etc.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in acomputer readable storage medium. Many of the features described hereinmay be implemented as processes that are specified as a set of programinstructions encoded on a computer readable storage medium. When theseprogram instructions are executed by one or more processing units, theycause the processing unit(s) to perform various operation indicated inthe program instructions. Examples of program instructions or computercode include machine code, such as is produced by a compiler, and filesincluding higher-level code that are executed by a computer, anelectronic component, or a microprocessor using an interpreter.

Through suitable programming, processing subsystem 102 can providevarious functionalities for system 100. For example, processingsubsystem 102 can execute an image visualization and analysisapplication 120 having a user interface which facilitates the review andinterpretation of scanned images of biological samples. The imagevisualization and analysis application 120 can provide variousfunctionality such as the ability to select user configurable options oruser selectable panels, or the ability to control navigation andannotate the images. In some embodiments, the analysis application 120includes logic such that only relevant user items (informative items,user selectable items, interactive items) are presented to the userbased upon whether a pre-established condition is met, e.g. whether azoom level meeting or exceeding a predefined threshold is selected bythe user or whether a certain type image is selected for review which isstained for the presence of a particular biomarker. In some embodiments,additional components may be incorporated into the system and softwareof the present disclosure, including those components identified inUnited States Patent Application Publication No. 2012/0320094, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

In some embodiments, the image visualization and analysis application120 incorporates various interoperating modules (e.g., blocks of code)that, when executed by one or more processors within the processingsubsystem 102, implement aspects of the interface operation. Forexample, the image visualization and analysis application 120 caninclude a content fetcher 122, a content renderer 124, a GUI renderer126, and a UI interpreter 128.

In some embodiments, content fetcher 122 can include instructions forinteracting with (e.g. accessing) a local database (e.g. storagesubsystem 104) or with network interface 112 to fetch or otherwiseretrieve content items, such as image data and/or subject data. In someembodiments, the content fetcher 122 is configured to access a pluralityof scanned images, each of the scanned images derived from a subjectsample, and where each of the scanned images may each be stained for thepresence of one or more biomarkers or for hematoxylin and eosin. In someembodiments, the content fetcher 122 is configured to retrieve subjectinformation, image metadata, case history information, etc. In someembodiments, content fetcher 122 can include instructions forinteracting with scanning device 110 such that image data may beacquired from one or more slides having a tissue sample stained for thepresence of one or more biomarkers.

In some embodiments, content renderer 124 can include instructions forinterpreting fetched content items from one or more sources and thenpopulating or delivering the rendered content to image placeholders orother representations generated by the GUI renderer 126. For example,content renderer 124 may populate one or more rendered representationswith image data retrieved from content fetcher 122 (see representation401 of FIG. 4A). In some embodiments, the content renderer 124 candeliver subject information to other GUI elements, such as one or moreviewer panels, or place the retrieved subject information into a portionof a GUI representation. In some embodiments, the content renderer 124may deliver metadata to other GUI elements, e.g. tissue type, stainsapplied, scanning parameters, z-stack layers, focus layers, etc. In someembodiments, content renderer 124 can also process the acquired imagedata, e.g., applying any pre-processing to acquired images.

In some embodiments, GUI renderer 126 creates graphical user interface(GUI) elements to be presented to the user along with the content itemsrendered by content renderer 124 or other system modules. GUI renderer126 can include code defining the location and appearance of GUIelements, such as a menu bar items and viewer panels, each of which maybe interactive elements themselves or which may include interactiveelements. In some embodiments, the GUI renderer 126, together withsignals received from the UI interpreter 128, may determine whethercertain menu bar items or viewer panels should be enabled or madeotherwise available to a user, such as depending on whether apre-established condition has been met, e.g. whether a threshold zoomlevel or magnification level has been selected. For example, a menu baritem may be activated by a user, whereby the user would then be allowedto select configuration options or panel views from a drop-down menu(see, e.g., FIGS. 5A and 5B). In some embodiments, the GUI renderer 126can incorporate acquired image data provided from the content fetcher122 or the content renderer 124 into some or all of the GUI elements(e.g. an actual image of a scanned biological sample may be displayedwithin a representation within the user interface, etc.).

By way of example, the GUI renderer 126 may generate a series ofrepresentations 401 which may be populated with image data retrieved bythe content fetcher 122. Examples of representations are illustrated inFIGS. 4A, 4B, and 4C. These representations may be interactiverepresentations. For example, if a user clicks on any particularrepresentation (e.g. representation 401 of FIG. 4A) (such as interpretedby UI interpreter 128), the GUI renderer 126 may update thecorresponding display in a viewer panel, such as a slide navigatorviewer panel.

Likewise, the GUI renderer 126 may generate a series of viewer panels.In some embodiments, the generated viewer panels are interactive panelswhere a user may select certain configurable options. For example, azoom panel may include a slider bar where a user can select a particularpreset zoom level, e.g. 1×, 2×, 10×, 40×, etc., or where the user caninput a particular zoom level value. In some embodiments, the viewerpanels are configured to convey relevant information to a user, e.g. acase log panel may provide a history of the user configurable selectionsmade by the user during analysis of the image data. In addition, the GUIrenderer 126 may render a visualization that shows hidden or items whichare unavailable or not enabled for a user to interact with.

UI interpreter 128 can receive user input, e.g., via a user input device106, and can interpret the input to determine actions to be performed bythe analysis application 120. For example, UI interpreter 28 candetermine which GUI element (e.g. an icon, or a selectable item in menu,context menu, dropdown list, buttons, a representation, etc.) the userselected and initiate the corresponding action (e.g., add an annotation,display additional content information, zoom to a selected zoom levelgenerate a report for exporting). For example, the UI interpreter 128may detect if a user selects an annotation tool (see annotation tools405 in FIGS. 4A and 4B) and may send a signal to the GUI renderer 126 todisplay additional user selected items. In some embodiments, theannotation tools include a manual region of interest (ROI) generationtool, an automated ROI generation tool, a tool which allows the drawingof a shape (e.g. an arrow), a measurement tool, or a text inputgeneration tool. Each of these tools may be independently disabled orhidden based on the context of user interactions. In some embodiments,menu items which may be selected include those that run certain imageprocessing algorithms, e.g. membrane detection algorithms, celldetection and counting algorithms, nucleus detection algorithms, scoringalgorithms, heat map generation algorithms, tissue masking algorithms,tissue type identification algorithms, etc. (see, for example, PCTPublication Nos. WO2016/120442 and WO2015/113895 and US PatentApplication Publication Nos. 2017/0154420, 2017/0372117, 2017/0103521,2017/0140246, 2015/0347702, 2017/0082627, 2014/0377753, 2017/0337695,2017/0323148 and 2017/0243051, the disclosures of which are herebyincorporated by reference herein in their entireties). Inputs receivedfrom the UI interpreter 128 may be used to determine whether apre-established condition has been met.

It will be appreciated that system 100 is illustrative and thatvariations and modifications are possible. Further, while system 100 isdescribed with reference to particular blocks, it is to be understoodthat these blocks are defined for convenience of description and are notintended to imply a particular physical arrangement of component parts.Further, the blocks need not correspond to physically distinctcomponents. Blocks can be configured to perform various operations,e.g., by programming a processor or providing appropriate controlcircuitry, and various blocks might or might not be reconfigurabledepending on how the initial configuration is accessed. Embodiments ofthe present disclosure can be realized in a variety of apparatusincluding electronic devices implemented using any combination ofcircuitry and software. Image visualization and analysis application 120is also illustrative, and specific implementations may include more orfewer modules than described herein. Moreover, while a particular modulemay be described as performing a particular function, such descriptionsare not intended to imply a particular function performed by the moduleor a particular set of instructions included within such module.

FIG. 2B depicts a client interface 140 in communication with a network130 and a system 100 (such as the systems depicted in FIGS. 1 and 2 ).The client interface 140 may be a stand-alone application (e.g.stand-alone image visualization and analysis software) or a web-browseror other interface software which allows remote access to imagevisualization and analysis application 120. For example, clientinterface 140 allows a remote operator to log-into system 100 (such asthe system depicted in FIGS. 1 and 2 ) and access stored image data(such as data stored in storage subsystem 104 or other network attachedstorage device) or image data uploaded to system 100 for processing. Insome embodiments, the client interface 140 may include any of thesoftware modules described herein. In this way, a remote user mayremotely interact with elements of the system (e.g. configurableelements) such that image data may be analyzed and/or interpreted (e.g.a histologist or pathologist may select user configurable parameters,such as menu bar tools and/or viewer panels.

In some embodiments, the graphical user interface is adapted to displayonly certain features depending upon the zoom level selected. Forexample, depending on the zoom level selected, certain tools within amenu bar may not enabled (see FIG. 4A). Likewise, certain panelspresented in proximity to the visualization of the accessed images maynot be made available. In general, the system 100 may be configured to“restrict” access to certain tools and panels that, given a certain zoomlevel, would not be contextually relevant during image analysis. Forexample, at certain zoom levels, annotating a specific portion of animage may not be feasible if there does not exist sufficient resolutionbetween certain features within the image. As such, if the softwaredetermines that a tool is not relevant at the select zoom level, thattool will not be enabled and, as noted above, this facilitates a quickerreview of the data presented within the visualizations and also providesfor an enhanced user experience and one free from potential confusion.

FIG. 3 sets forth a flowchart setting forth a method of visualizingimage data associated derived from a biological sample stained for thepresence of one or more biomarkers. At step 300, at least one image of abiological sample is accessed. In some embodiments, the biologicalsample is stained for the presence of one or more biomarkers. In someembodiments, multiple images are retrieved, such as multiple images fromthe same biological sample, but where each image comprises staineddenoting the presence or absence of a particular biomarker.

Subsequently, a first visualization is rendered within the graphicaluser interface (step 310), wherein the first visualization at leastincludes a rendering of the at least one image at a first zoom level. Insome embodiments, the first visualization 400 includes a rendering ofmultiple accessed images (see, e.g., FIG. 4A). In some embodiments, thefirst visualization includes a series of representations 401 (e.g. imageplaceholders) where each representation 401 may include one of theaccessed images 410A or 410B. In some embodiments, each representation401 is the same size and/or shape. In some embodiments, eachrepresentation 401 comprises a first portion 402 which includes one ofthe accessed images 410 and a second portion 403 which includesidentifying indicia. In some embodiments, the identifying indiciaincludes an identification of the stain appearing in the image or thebiomarker which was stained. By way of example, a second portion 403 ofrepresentation 401 indicates that the particular image 410A within thefirst portion 402 was stained with hematoxylin and eosin.

In some embodiments, a first series of tools (e.g. annotation tools 405)within a menu bar 404 are contemporaneously displayed with the firstvisualization 400 (step 320). As noted above, and as illustrated in FIG.4A, some of the tools are unavailable for selection, i.e. they are“greyed out” (see, e.g., setup calibration tool 408) while other toolsare available for selection and appearing as white icons (see, e.g.,rotation tool 407). By way of further example, each of the fiveannotation tools 405 are greyed out and thus disabled. In someembodiments, certain tools may be completely hidden from a user at aparticular zoom level as described further herein. As such, the firstseries of tools (i.e. those available for selection and/or representedas white icons) represents a subset of all of those tools that may beavailable to the user. Given the zoom level of the four images depictedin the representations 401 of FIG. 4A, it is believed that those toolswhich are not available, i.e. those that are “greyed out,” would not berelevant at the given zoom level of 1× as illustrated in FIG. 4A. Saidanother way, those tools that are not enabled or which are hidden fromuser selection are determined to be ineffective at the selected zoomlevel. For example, it is believed that there would be no value for thehistologist or pathologist to perform a measurement or to draw an arrowto a cell since, at the given zoom level (e.g. 1×), sufficient cellularfeatures would not be able to be resolved such that a measurement couldaccurately be made or that an arrow could correctly be placed pointingto a particular desired structure.

In some embodiments, a first set of panels may also be contemporaneouslydisplayed with the first visualization and with the first series oftools. For example, a zoom panel 406 and a slide navigator panel 409 maycontemporaneously be displayed with the first visualization and with thefirst series of tools. As with the first series of tools, only thosepanels that are contextually relevant at the given level of zoom arepresented. For example, FIG. 5A illustrates panels that may be selectedby a user in a contextual or dropdown menu. Those items that are notavailable are not able to be selected and are “greyed out.” Likewise,those panels that are able to be selected are shown in white and areable to be selected by the user. Moreover, those panels that havealready been selected may be marked with an indicia, e.g. a check mark.

Following the contemporaneous presentation of the first visualization(e.g. the three representations 401 in FIG. 4A) and of the first seriesof tools (those that are not “greyed out” in menu bar 404), and/or thefirst set of panels, a user may then interact with the visualizationsuch as to change the zoom level, e.g. to increase the zoom of one ormore of the accessed images presented in the first visualization,enabling the user to view at least a portion of one of the accessedimages in greater detail, and thereby providing at least a secondvisualization at a second zoom level 430. By “increasing the zoom” or“zooming into the image” it is meant that a portion of the image ismagnified thereby increasing the visual resolution of that portion ofthe image. For example, accessed image 410B is presented in FIG. 4A at a1× zoom level, and it will be appreciated that at this zoom levelcertain tissue structures, e.g. 420A, are difficult to decipher. When,however, the zoom level of image 410B is increased as shown in FIG. 4B(e.g. increasing the zoom level from 1× to about 10×), the magnificationand/or resolution of the tissue structure 420B is shown in greaterdetail, e.g. at a level where individual cells may be resolved.

Contemporaneous with the display of the second visualization (step 330)at the second zoom level 430, a second series of tools in a menu bar 404are displayed (step 340). Like the first series of tools, the secondseries of tools again represents a subset of all of those toolsavailable to a user. By way of example, and in comparison, to the firstseries of tools available (see FIG. 4A), the second series of toolsincludes each of the five annotation tools 405 (see FIG. 4B).

In the particular embodiment illustrated in FIG. 4B, the second set oftools is inclusive of the first series of tools, i.e. the second set oftools includes all of those tools available in the first series. In someembodiments, the second set of tools does not include all of the toolsprovided in the first series. In some embodiments, the second series oftools comprises at least one different tool than that provided in thefirst series of tools.

In some embodiments, a second set of panels are displayedcontemporaneously with the second visualization at the second zoom level430 and with the second series of tools. As with the second series oftools, only those panels that are contextually relevant at the givenlevel of zoom are presented. In the embodiment illustrated in FIG. 4B,and as compared with the embodiment of FIG. 4A, a slide panel 425 isautomatically displayed at the about 10× zoom level. Turning to FIG. 5B,once again those items that are not available are not able to beselected and are “greyed out.” Likewise, those viewer panels that areable to be selected are shown in white and are able to be selected bythe user. Moreover, those panels that have already been selected aremarked with an indicia, e.g. a check mark.

FIG. 4C illustrates the selection of a zoom level (e.g. an intermediatezoom level) that falls between the zoom level depicted in FIG. 4A andthe zoom level depicted in FIG. 4B. While four representations 401 wereillustrated in FIG. 4A at a zoom level of 1×, at the zoom level in FIG.4C, only three representations are visualized, and only one of the threerepresentations is fully visualized. Notably, the same tools appear asavailable in FIG. 4C as in FIG. 4A. Likewise, the same panels appear inFIG. 4C as in FIG. 4A. As such, FIG. 4C illustrates that a thresholdzoom value has not been reached by the user such that the imagevisualization and analysis application 420 would make available othertools in menu bar 404 or, for that matter, make available other viewerpanels available. In this regard, FIG. 4C illustrates that itemsdisplayed and made available to a user are contextually dependent, herecontextually dependent upon the level of zoom selected by a user.

In some embodiments, each of the menu items and/or viewer panels have apre-programmed threshold zoom level that must be attained prior toenabling the respective menu items and/or viewer panels available. Forexample, with reference to FIG. 4A, the annotation tools 405 may onlybecome available when a user selects a zoom level which exceeds apre-determined threshold zoom level value, e.g. 5×. In some embodiments,each individual tool within the menu bar 404 may have a differentpre-determined threshold value. For example, a first annotation tool mayhave a predetermined threshold value of 2×, a second annotation tool mayhave a predetermined threshold value of 6×, and a slide calibration toolmay have a predetermined threshold value of 10×. In some embodiments,the threshold zoom value may depend on a tissue type or a stain beingobserved.

In some embodiments, the threshold for whether to make available a menubar items or a viewer panel does not need to be tied to a predeterminedthreshold zoom value, i.e. a value of 1×, 2×, 4×, 8×, 16×, 32×, etc.Rather, the threshold can be tied to whether a zoom level is selectedwhere individual cells or individual nuclei may be resolved by a user.Alternatively, the threshold can be tied to whether a zoom level isselected where a certain number of cells are present a predefined area(pixel×pixel), e.g. 100 cells in a 500 pixel×500 pixel area. In someembodiments, different elements may be visualized depending on theavailable display resolution. For example, the threshold may bepredefined as “p” if a display resolution is “m×n,” but may bepredefined as “p*q” if the display resolution is (m*q×n*q), where q is ascaling factor to account for differences in display resolutions. In yetother embodiments, whether certain viewer panels may be displayed orhidden may be tied to an available display resolution. For example,although 5 viewer panels may be made available or unhidden when athreshold zoom level is reached (i.e. they become contextually relevantand made available), the system may continue to hide certain panels ifavailable screen “real estate” is not available due to low or limiteddisplay resolution, and the system may determine which of the availableviewer panels are most relevant given the display resolution limitationsand prioritize those panels for display. In addition, and by way ofexample, the viewer panels could be cycled as needed to accommodate forlimited display resolutions.

As noted herein, in some embodiments, entire menu items may be hiddenuntil a particular zoom level is selected by the user. For example,assume that a menu bar were to include items A, B, C, D, E, F, G, and H.Further assume, that a zoom level of 1×, that only menu bar items A, B,E, and H are shown within a graphical user interface, as depicted inFIG. 6A. According to the present disclosure, those menu bar items A, B,E, and H are the only tools relevant and useful for a user at a zoomlevel of 1×. Once the user zooms into one of the images (see FIG. 6B),additional menu bar items may be shown, e.g. menu bar items C and G,provided that menu bar items (i.e. C and G) are relevant to the level ofzoom selected (e.g. 3×). Once the user further zooms into one of theimages (see FIG. 6C), here 601B, yet additional menu bar items may beshown, i.e. D and F, again provided that those yet additional menu baritems are relevant at the level of zoom selected (e.g. 8×). FIG. 6Cfurther illustrates that a second viewer panel, i.e. “Viewer Panel 2,”becomes shown by the graphical user interface, but again only when aparticular zoom level is reached (i.e. at least a zoom level greaterthan 3×, such as at 8×).

In some embodiments, an input image is received by the visualizationsystem and visualizations are provided at a default zoom level, e.g. azoom level of 1×. In some embodiments, the system receives a user input,e.g. a selection of an updated zoom level, a selection of a particularimage. In some embodiments, a comparison is made between the receiveduser input and threshold conditions, e.g. a comparison is made between areceived user input of a zoom level to determine if a threshold zoomlevel is attained or a comparison is made between a received user inputof an image selection to determine the biomarkers identified in theimage. In some embodiments, if a threshold zoom level is met, thenadditional visualization elements (e.g. tools, panels) may be presentedto the user. In some embodiments, if different biomarkers are selectedin a second image as compared with a first image, then additionalvisualization elements may be presented to the user. In someembodiments, at least one GUI element is changed based on a userselection.

Additional Embodiments

In some embodiments, each displayed visualization or element has aposition within a coordinate system of a display provided within aninterface application, such as a browser. For example, icons for tools,such as image analysis tools, have a position within a displaycoordinate system. For instance, if a display has a resolution of4,000×3000 and each pixel is considered a point in a coordinate system,then an icon for an annotation tool may have a position defined by anarea of pixels [150, 200] (top left corner), [160, 200] (top rightcorner), [150, 210] (bottom left corner), and [160, 210] (bottom rightcorner). In some embodiments, viewer panels, image data, and otherrepresentations may have positions within the coordinate system of thedisplay. In some embodiments, a first viewer panel may have a firstpositional area while a second viewer element may have a second positionarea. In some embodiments, each viewer panel may have a variablepositional area depending on the type of information being displayed andthe quantity of information available. In some embodiments, thepositions of various elements within the coordinate system of thedisplay may be fixed or may be variable depending upon context.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Any of the modulesdescribed herein may include logic that is executed by the processor(s).“Logic,” as used herein, refers to any information having the form ofinstruction signals and/or data that may be applied to affect theoperation of a processor. Software is an example of logic.

A computer storage medium can be, or can be included in, acomputer-readable storage device, a computer-readable storage substrate,a random or serial access memory array or device, or a combination ofone or more of them. Moreover, while a computer storage medium is not apropagated signal, a computer storage medium can be a source ordestination of computer program instructions encoded in an artificiallygenerated propagated signal. The computer storage medium can also be, orcan be included in, one or more separate physical components or media(e.g., multiple CDs, disks, or other storage devices). The operationsdescribed in this specification can be implemented as operationsperformed by a data processing apparatus on data stored on one or morecomputer-readable storage devices or received from other sources.

The term “programmed processor” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable microprocessor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus also can include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,subprograms, or portions of code). A computer program can be deployed tobe executed on one computer or on multiple computers that are located atone site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random-access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., an LCD (liquid crystal display), LED(light emitting diode) display, or OLED (organic light emitting diode)display, for displaying information to the user and a keyboard and apointing device, e.g., a mouse or a trackball, by which the user canprovide input to the computer. In some implementations, a touch screencan be used to display information and receive input from a user. Otherkinds of devices can be used to provide for interaction with a user aswell; for example, feedback provided to the user can be in any form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input. In addition, a computer can interactwith a user by sending documents to and receiving documents from adevice that is used by the user; for example, by sending web pages to aweb browser on a user's client device in response to requests receivedfrom the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example,the network 20 of FIG. 1 can include one or more local area networks.

The computing system can include any number of clients and servers. Aclient and server are generally remote from each other and typicallyinteract through a communication network. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

All of the U.S. patents, U.S. patent application publications, U.S.patent applications, foreign patents, foreign patent applications andnon-patent publications referred to in this specification and/or listedin the Application Data Sheet are incorporated herein by reference, intheir entirety. Aspects of some embodiments can be modified, ifnecessary to employ concepts of the various patents, applications andpublications to provide yet further embodiments.

Although the present disclosure has been described with reference to anumber of illustrative embodiments, it should be understood thatnumerous other modifications and embodiments can be devised by thoseskilled in the art that will fall within the spirit and scope of theprinciples of this disclosure. More particularly, reasonable variationsand modifications are possible in the component parts and/orarrangements of the subject combination arrangement within the scope ofthe foregoing disclosure, the drawings, and the appended claims withoutdeparting from the spirit of the disclosure. In addition to variationsand modifications in the component parts and/or arrangements,alternative uses will also be apparent to those skilled in the art.

1. A method comprising: accessing a plurality of images of a biologicalsample stained for presence of one or more biomarkers, wherein thebiological sample comprises a plurality of cells; displaying, on adisplay screen, a graphical user interface in a first state, thegraphical user interface in the first state comprising the plurality ofimages at a first magnification level, a first set of menu items in amenu bar, and a first viewer panel; receiving a user input to change azoom level of an image of the plurality of images from the firstmagnification level to a second magnification level; and in response toreceiving the user input to change the zoom level of the image of theplurality of images from the first magnification level to the secondmagnification level, displaying, on the display screen, the graphicaluser interface in a second state, the graphical user interface in thesecond state comprising the image of the plurality of images at thesecond magnification level, a second set of menu items in the menu bar,and the first viewer panel.
 2. The method of claim 1, wherein at leastone menu item in the first set of menu items is included in the secondset of menu items.
 3. The method of claim 1, wherein a number of menuitems in the second set of menu items is greater than a number of menuitems in the first set of menu items.
 4. The method of claim 1, whereinthe second magnification level is greater than the first magnificationlevel.
 5. The method of claim 1, further comprising: receiving a userinput to change the zoom level of the image of the plurality of imagesfrom the second magnification level to a third magnification level; andin response to receiving the user input to change the zoom level of theimage of the plurality of images from the second magnification level tothe third magnification level, displaying, on the display screen, thegraphical user interface in a third state, the graphical user interfacein the third state comprising the image of the plurality of images atthe third magnification level, a third set of menu items in the menubar, and a second viewer panel.
 6. The method of claim 5, wherein anumber of menu items in the third set of menu items is greater than thenumber of menu items in the second set of menu items.
 7. The method ofclaim 5, wherein the third magnification level is greater than thesecond magnification level.
 8. A system comprising: a processor; and amemory storing instructions which, when executed by the processor, causethe processor to perform operations comprising: accessing a plurality ofimages of a biological sample stained for presence of one or morebiomarkers, wherein the biological sample comprises a plurality ofcells; displaying, on a display screen, a graphical user interface in afirst state, the graphical user interface in the first state comprisingthe plurality of images at a first magnification level, a first set ofmenu items in a menu bar, and a first viewer panel; receiving a userinput to change a zoom level of an image of the plurality of images fromthe first magnification level to a second magnification level; and inresponse to receiving the user input to change the zoom level of theimage of the plurality of images from the first magnification level tothe second magnification level, displaying, on the display screen, thegraphical user interface in a second state, the graphical user interfacein the second state comprising the image of the plurality of images atthe second magnification level, a second set of menu items in the menubar, and the first viewer panel.
 9. The system of claim 8, wherein atleast one menu item in the first set of menu items is included in thesecond set of menu items.
 10. The system of claim 8, wherein a number ofmenu items in the second set of menu items is greater than a number ofmenu items in the first set of menu items.
 11. The system of claim 8,wherein the second magnification level is greater than the firstmagnification level.
 12. The system of claim 8, the operations furthercomprising: receiving a user input to change the zoom level of the imageof the plurality of images from the second magnification level to athird magnification level; and in response to receiving the user inputto change the zoom level of the image of the plurality of images fromthe second magnification level to the third magnification level,displaying, on the display screen, the graphical user interface in athird state, the graphical user interface in the third state comprisingthe image of the plurality of images at the third magnification level, athird set of menu items in the menu bar, and a second viewer panel. 13.The system of claim 12, wherein a number of menu items in the third setof menu items is greater than the number of menu items in the second setof menu items.
 14. The system of claim 12, wherein the thirdmagnification level is greater than the second magnification level. 15.A non-transitory computer-readable medium storing computer-readableinstructions that, when executed by a processor, cause the processor toperform operations including: accessing a plurality of images of abiological sample stained for presence of one or more biomarkers,wherein the biological sample comprises a plurality of cells;displaying, on a display screen, a graphical user interface in a firststate, the graphical user interface in the first state comprising theplurality of images at a first magnification level, a first set of menuitems in a menu bar, and a first viewer panel; receiving a user input tochange a zoom level of an image of the plurality of images from thefirst magnification level to a second magnification level; and inresponse to receiving the user input to change the zoom level of theimage of the plurality of images from the first magnification level tothe second magnification level, displaying, on the display screen, thegraphical user interface in a second state, the graphical user interfacein the second state comprising the image of the plurality of images atthe second magnification level, a second set of menu items in the menubar, and the first viewer panel.
 16. The non-transitorycomputer-readable medium of claim 15, wherein at least one menu item inthe first set of menu items is included in the second set of menu items.17. The non-transitory computer-readable medium of claim 15, wherein anumber of menu items in the second set of menu items is greater than anumber of menu items in the first set of menu items.
 18. Thenon-transitory computer-readable medium of claim 15, wherein the secondmagnification level is greater than the first magnification level. 19.The non-transitory computer-readable medium of claim 15, the operationsfurther comprising: receiving a user input to change the zoom level ofthe image of the plurality of images from the second magnification levelto a third magnification level; and in response to receiving the userinput to change the zoom level of the image of the plurality of imagesfrom the second magnification level to the third magnification level,displaying, on the display screen, the graphical user interface in athird state, the graphical user interface in the third state comprisingthe image of the plurality of images at the third magnification level, athird set of menu items in the menu bar, and a second viewer panel. 20.The non-transitory computer-readable medium of claim 19, wherein anumber of menu items in the third set of menu items is greater than thenumber of menu items in the second set of menu items.